Assessing by design—constructing a marking guide

NEED Improving assessment is a key aim of the Australian Council of Deans of Science (ACDS). In 2023, it funded workshops to help improve assessment across the sciences, where science educators across Australia worked with assessment experts from the Australian Council of Educational Research and discipline leaders to learn how to write, review and undertake psychometric analysis of assessment items. Effective marking guides (rubrics) are an essential part of aligning assessment to learning outcomes, showing the performance expectations of an assessment, and demonstrating the objectivity of the assessment’s marking. Designing and developing marking guides (rubrics) is both an art and a skill. There is literature, but there is no replacement for experience and for adopting a design approach to constructing excellent marking guides. It is worth spending time on the design and writing of a marking guide as it will likely improve marking both through improved feedback to students, discrimination of performance, reliability and through increasing marking efficiency. WHAT WE WILL DO In this session we will work through three stages to develop skills at constructing marking guides: We will work together to identify the key determinants of a good marking guide. This discussion will be supported by consideration of the literature and of our collective experience. We will apply those determinants to working collaboratively to develop and refine a marking guide as a group. We will work in small groups to refine marking guides brought to the workshop by participants. WHO SHOULD ATTEND All are welcome to join us for this workshop. It follows on from the assessment workshops sponsored by the ACDS in 2023 and attended by over thirty academics around Australia. WHAT TO BRING We encourage participants to bring a marking guide that they would like to improve

Strengthening the global system of protected areas post-2020: A perspective from the IUCN World Commission on Protected Areas

Protected areas are the cornerstones of biodiversity conservation and have never been more relevant than at the present time when the world is facing both a biodiversity and a climate change crisis. The International Union for Conservation of Nature (IUCN) World Commission on Protected Areas (WCPA) has been helping to set global standards and best practice guidelines in protected area planning and management for 60 years. Following this guidance, many countries have made significant progress toward their Aichi Target 11 commitments under the Convention on Biological Diversity (CBD). The global community will be coming together at the 15th Conference of the Parties of the CBD to set new biodiversity conservation targets for the next decade, as milestones to 2050 and a vision of “a world living in harmony with nature.” This paper lays out the WCPA perspective on priorities for supporting effective protected and conserved areas for the post-2020 era

Building the capacity of academics to assess higher order skills through improved assessment design

Across the sciences, high-stakes assessment (such as end-of-semester exams) often consist largely, if not solely, of multiple choice questions (MCQs). MCQs have advantages in large-scale testing, including automated marking and high reliability (Haladyna 2004). But in biomedical education, MCQs usually test knowledge recall (the lowest “Remembering” category in the cognitive domain of Blooms taxonomy). Psychometric analysis of MCQs exams in biochemistry and physiology from four universities has shown that they contained a substantial number of questions that were too easy for the cohort and lacked questions testing higher-order skills such as problem-solving, and knowledge analysis and application . It is possible to design MCQs that test higher-order cognitive processes—the UMAT and GAMSAT medical entrance exams routinely include MCQs which test higher-order cognitive skills. But to write questions of this standard requires skill and effort. The objectives of this workshop are: To build academic capability to design and write MCQ exams that test students’ capacity to solve problems and apply their knowledge/competencies in new contexts; and to improve the validity of assessment by adopting a rational assessment design approach, including mapping of questions against subject learning objectives

BACK TO THE FUTURE: FACILITATING ASSESSMENT DESIGN, COLLABORATION, AND OUTCOMES ALIGNMENT WITH AN ASSESSMENT FRAMEWORK

Aligning assessment questions to content and process variables is a collaborative activity. Robust practice requires participants to first make an individual judgement about the alignment, and then share their judgements and adjust them to work towards building consensus. In this workshop, attendees will participate in an interactive activity using the Australian Council for Educational Research's Swift platform to judge exam questions against the content and process variables of an assessment framework and then build consensus as a group. This will inform academics how they can use the framework in their own practice and within their own institution. People from any disciplinary background are welcome to join us. We will be using life science example questions, but it will only be necessary to understand the intent of the question, not be able to answer it

Usage and acceptability of the ibobbly App: A pilot trial for suicide prevention in Aboriginal and Torres Strait Islander youth

Background: The proliferation of mental health apps purporting to target and improve psychological wellbeing is ever-growing and also concerning: Few apps have been rigorously evaluated, and, indeed, the safety of the vast majority of them has not been determined. Over 10,000 self-help apps exist but most are not used much after being downloaded. Gathering and analyzing usage data and the acceptability of apps are critical to inform consumers, researchers, and app developers. Objective: This paper presents pilot usage and acceptability data from the iBobbly suicide prevention app, an app distributed through a randomized controlled trial. Methods: Aboriginal and Torres Strait Islander participants from the Kimberley region of Western Australia completed a survey measuring their technology use in general (n=13), and data on their experiences with and views of the iBobbly app were also collected in semistructured interviews (n=13) and thematically analyzed. Finally, engagement with the app, such as the number of sessions completed and time spent on various acceptance-based therapeutic activities, was analyzed (n=18). Both groups were participants in the iBobbly app pilot randomized controlled trial (n=61) completed in 2015. Results: Regression analysis indicated that app use improved psychological outcomes, although only minimally, and effects were not significant. However, results of the thematic analysis indicated that the iBobbly app was deemed effective, acceptable, and culturally appropriate by those interviewed. Conclusions: There is a scarcity of randomized controlled trials and eHealth interventions in Indigenous communities, while extremely high rates of psychological distress and suicide persist. In this environment, studies that can add evidence from mixed-methods approaches are important. While the regression analysis in this study did not indicate a significant effect of app use on psychological wellbeing, this was predictable considering the small sample size (n=18) and typically brief app use. The results on engagement with the iBobbly app were however positive. This study showed that Indigenous youth are early and frequent users of technology in general, and they regarded the iBobbly app to be culturally safe and of therapeutic value. Qualitative analyses demonstrated that iBobbly app use was associated with self-reported improvements in psychological wellbeing, mental health literacy, and reductions in shame. Importantly, participants reported that they would recommend other similar apps if available to their peers

Is self-guided internet-based cognitive behavioural therapy (iCBT) harmful? : An individual participant data meta-analysis

BACKGROUND: Little is known about potential harmful effects as a consequence of self-guided internet-based cognitive behaviour therapy (iCBT), such as symptom deterioration rates. Thus, safety concerns remain and hamper the implementation of self-guided iCBT into clinical practice. We aimed to conduct an individual participant data (IPD) meta-analysis to determine the prevalence of clinically significant deterioration (symptom worsening) in adults with depressive symptoms who received self-guided iCBT compared with control conditions. Several socio-demographic, clinical and study-level variables were tested as potential moderators of deterioration. METHODS: Randomised controlled trials that reported results of self-guided iCBT compared with control conditions in adults with symptoms of depression were selected. Mixed effects models with participants nested within studies were used to examine possible clinically significant deterioration rates. RESULTS: Thirteen out of 16 eligible trials were included in the present IPD meta-analysis. Of the 3805 participants analysed, 7.2% showed clinically significant deterioration (5.8% and 9.1% of participants in the intervention and control groups, respectively). Participants in self-guided iCBT were less likely to deteriorate (OR 0.62, p < 0.001) compared with control conditions. None of the examined participant- and study-level moderators were significantly associated with deterioration rates. CONCLUSIONS: Self-guided iCBT has a lower rate of negative outcomes on symptoms than control conditions and could be a first step treatment approach for adult depression as well as an alternative to watchful waiting in general practice

Efficacy of Self-guided Internet-Based Cognitive Behavioral Therapy in the Treatment of Depressive Symptoms : A Meta-analysis of Individual Participant Data

IMPORTANCE Self-guided internet-based cognitive behavioral therapy (iCBT) has the potential to increase access and availability of evidence-based therapy and reduce the cost of depression treatment. OBJECTIVES To estimate the effect of self-guided iCBT in treating adults with depressive symptoms compared with controls and evaluate the moderating effects of treatment outcome and response. DATA SOURCES A total of 13 384 abstracts were retrieved through a systematic literature search in PubMed, Embase, PsycINFO, and Cochrane Library from database inception to January 1, 2016. STUDY SELECTION Randomized clinical trials in which self-guided iCBT was compared with a control (usual care, waiting list, or attention control) in individuals with symptoms of depression. DATA EXTRACTION AND SYNTHESIS Primary authors provided individual participant data from 3876 participants from 13 of 16 eligible studies. Missing data were handled using multiple imputations. Mixed-effects models with participants nested within studies were used to examine treatment outcomes and moderators. MAIN OUTCOMES AND MEASURES Outcomes included the Beck Depression Inventory, Center for Epidemiological Studies-Depression Scale, and 9-item Patient Health Questionnaire scores. Scales were standardized across the pool of the included studies. RESULTS Of the 3876 study participants, the mean (SD) age was 42.0 (11.7) years, 2531 (66.0%) of 3832 were female, 1368 (53.1%) of 2574 completed secondary education, and 2262 (71.9%) of 3146 were employed. Self-guided iCBT was significantly more effective than controls on depressive symptoms severity (ß =-0.21; Hedges g = 0.27) and treatment response (ß = 0.53; odds ratio, 1.95; 95% CI, 1.52-2.50; number needed to treat, 8). Adherence to treatment was associated with lower depressive symptoms (ß =-0.19; P =.001) and greater response to treatment (ß = 0.90; P <.001). None of the examined participant and study-level variables moderated treatment outcomes. CONCLUSIONS AND RELEVANCE Self-guided iCBT is effective in treating depressive symptoms. The use of meta-analyses of individual participant data provides substantial evidence for clinical and policy decision making because self-guided iCBT can be considered as an evidence-based first-step approach in treating symptoms of depression. Several limitations of the iCBT should be addressed before it can be disseminated into routine care

The effect of web based depression interventions on self reported help seeking: randomised controlled trial [ISRCTN77824516]

BACKGROUND: To date, there has been very little work investigating behaviour changes induced by interventions that are designed to increase help seeking. The present paper examines the effects of two Internet depression websites on help seeking. METHODS: 414 individuals with elevated scores on a depression assessment scale were randomly allocated to a depression information website, a cognitive-behavioural skills training website (CBT) or an attention control condition. Reports of help seeking for specific treatments, from specific sources and for categories of treatments were assessed. RESULTS: Relative to the control, the depression information site was associated with decreases in seeking support from friends and family, the use of music and of everyday treatments and no increase in seeking evidence based interventions. The CBT site was associated with the report of help seeking for CBT, massage and exercise. CONCLUSION: Methods to encourage the use of evidence-based treatments need further research to determine whether the assistance sought is evidence based and whether there are unintended effects